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FOREWORD 


This report volttme describes the analytic methodology and model development of a 
computer program for probabilistic, optimal assignment of launch vehicles and 
related program elements to advanced space missions. This study is being perform'e( 
for the National Aeronautics and Space; Administration imder Contract NAS 2-5202. 

The study is monitored by Mr. R. E. Slye and Mr. Harold Hornby of the Advanced 
Concepts and Missions Division of the Office of Advanced Research and Technology. 

Individuals of Lockheed Missiles & Space Company, Sunnyvale, California, who 
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contributed to this study are L. P. Fox, project leader; C. J. Golden, key technical 
member; and W. T. Lew. 



ABSTRACT 


The optimal, least cost assignment of laimch vehicles and related program elements 
for a total space program over an extended time period requires the solution of a large 
combinatorial problem. Using an accelerated search technique, this problem was 
solved in prior, work on a deterministic basis. The conversion of this comprehensive 
space program evaluation tool to a probabilistic model is the primary objective and re- 
sult of this study. The developed computer model retains the capability for determinis- 
tic solutions but adds a new, powerful dimension for probabQistlc evaluations and sen- 
sitivity analyses. With statistical data input, the program can output program costs 
quantified to any degree of certainty. The input/output structure is versatile with 
multiple options to adapt the program to the needs of the analyst. Using one option, 
output results can be smoothed under variable year-to-year budget constraints which 
reflect external economic conditions including growth and inflation. While applied to 
space systems in this study, the developed technique and basic model can be adapted 
to other optimal assignment problems by particularizing the parameters to the new 
problem. 
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SUMMARY 


This document is Volume 1 of a two volume report entitled Probabilistic Systems 
Modeling and Cost/Performance Methodologies for Optimization of Vehicle Assignment. 
Volume 2 provides details on the computer program developed during this study. This 
volume provides a technical description of an analysis of historical data, the develop- 
ment of the analytical approach, and the capabilities of the developed computer model. 

Building on a deterministic model developed under prior work; the conversion of this 
comprehensive space program evaluation tool to a probabilistic model is the primary 
objective and the major result of this study. Using a significantly modified branch- 
and -bound technique for accelerated search, the model evaluates the data from all 
combinations of launch vehicles and other interrelated space program elements, and 
selects a global optimum, least cost total space program for a specific mission profile. 
The model has multiple input and output options responsive to the needs of the analyst; 
these include stage matching to form vehicles, performance and time availability for 
vehicle -to -mission screening, and budget smoothing under -various ceilings and exter- 
nal economic conditions. Expendable, partially reusable, and fully reusable vehicles 
may be evaluated in the same program mix. 

Analysis of historical data clearly shows there are predictable cost uncertainties in 
data which apply to advanced space systems having technological risk. The log-normal 
distribution was identified as most appropriate for the anal 3 d:ic relationships in this 
model. The model retains the capability for deterministic solutions but adds the new, 
powerful -dimension of probabilistic evaluations and sensitivity analyses, which quantify 
the cost uncertainties known to exist in high technology, advanced space programs. 

The model can be applied to a wide range of space program evaluations — to macro - 
problems which evaluate various options of total space programs, to intermediate 
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problems which analyze separate portions of a space program (e. g. , optimizing a 
scientific, exploratory, service satellite program within a total space program), and 
to micro-problems (e.g. , determining the cost optimal subsystem among several 
alternates for a given space vehicle). ‘ Inwall cases the impact of cost imcertainty can 
be quantitatively assessed. 

In this application the optimal assignment model is applied to space systems. However, 
the technique can be readily applied to diverse optimal assignment problems by particu- 
larizing the parameters to the new problem. 
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Section 1 
INTRODUCTION 


1.1 BACKGROUND 

During a preceding phase of work under contract NAS2 -5202 (directed by the NASA 
Advanced Concepts and Missions Division) a deterministic computer model was devel- 
oped for the analysis and optimal assignment of launch vehicles and other program ele- 
ments that comprise a multi-mission space program over an emended period of time. 
This analytical tool quantitatively evaluates the many interrelated factors which enter 
such a large-scale problem. The computer model incorporates optional performance 
subroutines that determine vehicle-to-mission compatibility, evaluate the capability 
and time availability of supporting elements (such as launch sites and pads), and, most 
important, employ an accelerated search technique to determine optimal solutions based 
on least cost. 

The numerous functional options readUy selectable by the user provide flexibile adap- 
tion of the model to the needs of the analyst. A capability to output optimal assignments 
that are smoothed under parameterized economic conditions including variable budget 
ceilings, with growth and inflation factors, is one option available. Using these options, 
various candidates may be evaluated for space transportation systems including fulljj 
reusable, partially reusable, and expendable systems as elements of a total space pro- 
gram mix. 

Additional details on this deterministic model are provided in Ref. 1. 

1.2 OBJECTIVE AND TASKS 

Prior analyses of advanced systems data have shown that there is considerable varia- 
bility between planning cost estimates and costs actually experienced in developing new 



systems and making them operational. This variance is particularly notable in high 
technology systems involving state of art advances. 

Within limitations the impact of relatively simple variability in input data can be 
analyzed by iterative means using a deterministic model. However experience using 
this approach has shown that determination of the imcertalnties involved in the evalua- 
tion and the optimization of realistic advanced programs requires a very large nxmiber 
of iterations. Further, the dependence between entering variables, as detailed in sec- 
tion 3.3, has significant effect on solution outcomes but cannot be handled on a deter- 
ministic basis. 

The ‘objective of the present phase of effort, therefore, has been the conversion of the 
deterministic assignment and budget smoothing model to a probabilistic model. The 
converted model is to accept probabilistic inputs, perform internal analysis and op- 
timization, and provide outputs with their associated measures of certainty. 

During this phase of effort the above objective has been accomplished by effort under 
three basic tasks. A brief description of these tasks follows. 

1.2.1 Statistical Analysis of Data 

Historical data were analyzed on a preliminary basis to determine statistical charac- 
teristics of the data. Results of this analysis provided a basis for a tractable solution 
to a problem of this magnitude (i. e. , the feasibility of an analytic, probability density 
function approach rather than one using random numbers, e.g. , Monte Carlo). This 
analysis also determined preliminary values for statistical parameters which may be 
used in computer program input. 

1.2.2 Anal 3 rtic Approach 

Using the results of the preceding task an analytic approach was developed which solved 
this large scale problem while maintaining relatively short computer run times and 
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remaining within computer storage constraints. The logarithmic normal (log-normal) 
probability density function was identified for use in the analjdic relationships. This 
density function possesses the characteristics required by. such a cost growth distri- 
bution function and, since the log-normal distribution is functionally related to the 
normal distribution, statistical relationships of interest to the user are easily derived 
and computed. 

Using the log-normal distribution, anal3dic relationships were developed for use in the 
probabilistic model.' From the input of most likely and upper -tail values, expected 
values are derived for use by the optimizing algorithm. The statistical parameters 
involved in each assignment, including the dependence between these variables, are 
used to determine uncertainties associated with each assignment’, and probabUistie 
relationships- between- assignments. ■ ■ ; 

1.2.3 Computer Program Development ■ - • . 

In developing the probabilistic computer model, the deterministic model described in 
section 1. 1 was embedded within the new model which includes the analytic relation- 
ships developed above. Short rim times and minimum storage requirements were 
maintained, permitting production use of the program in a multi-user computer system. 
Because considerable emphasis was placed on flexibility, selectable options are avail- 
able to match the model to varying needs of the user. Some of the options which gen-- 
erate this flexibility are as follows: 

• Optimized assignment (least total cost which- meets mission requirements) 
of space program elements, e.g. , stage hardware, launch sites and pads, 
stage Integration, reusables, and others. Alternately a predetermined 
assignment may be input and smoothed if desired. 

• Optimization with or without budget smoothing under year-by-year budget 
constraints. 

o Variable economic conditions — budget ceilings, support bases, inflation — 
may be imposed over an input period of time. 
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• Assembly of vehicles from input stages considering physical and performance 
parameters (both variable); evaluation of alternative components on same stage. 

• For reusable stages, a given number to be procured may. be input or the model 
will compute the required number based on actual launch rates. 

e For reusables, capability for payload to be delivered in a single or multiple 
launches (modularization). 

• Several levels of vehicle-to-mission screening are available. The most com- 
prehensive includes type of stabilization, man-rating, number of required 
restarts, laimch site, and payload vs. requirements. 

• Length of space program selectable, from 1 to 20 years. 

• The development period may be stretched or accelerated during the smoothing 
process at the option of the user. Otherwise the input nominal development 
period is maintained. 

• Recurring cost dependence on learning rate may be selected from two types 
in wide use or bypassed as desired. 

• The deterministic model is available by leaving out all probabilistic input. 

All calculations based on these input data are then bypassed automatically. 

The wide range of input acceptable to this model and the wide range of output available 
from the model make it particularly useful as an evaluation tool. Some of these features 
are listed below. 

• Evaluating expendable, partially reusable, or fully reusable stage s/vehicles 
mixed in the same total space program. 

• Determines whether a stage/vehicle, launch pad, stage integration, etc. has 
been developed and is available when needed; if not, provides for development 
on a timely basis if feasible. 

• Pro-ration of all types of costs when there is multi-mission use of a program 
element (i.e. , family or shared costs). 

• Output of program cost based on most probable (modal) values, on expected 
values, or on values such that the residual probability of cost overrun is less 
than xx%. 
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o Output includes probability that the total cost of space program A will be ^ 
cost of program B, so comparisons may be made be^een competing assign- 
ments. For the optimal assignment, mean, modal; and 50% uncertainty inter- 
vals are plotted for each year’s spending level. 

More complete details on input options and other program adaptations are provided in 
Section 4 and in Appendix A (Vol. 2). 

1.3 GENERAL MODEL DESCRIPTION 

As summarized, the probabilistic model solves the same large scale combinatorial 
problem, including multiple interrelated program elements, as the previous determin- 
istic model (Ref. 1). For example, finding the. optimal assignment of 25 candidate 
vehicles to 500 missions over a 20-year period leads to a problem having possible 
solutions on the order of 25^^^. lii addition, the new model handles the various cost 
factors on a probabilistic basis. As before, factors are handled' explicitly , so a global 
optimum solution is assured. The major difference between the two models is that in- 
put costs, based on a statistical analysis of historical 'data, are "entered as parameters 
of probability distributions, are appropriately analyzed on this basis within the model, 
and results are output with solution certainties (or uncertainties as desired) quantita- 
tively specified. 

In addition to this large scale optimal assignment problem .the model analyzes many 
other important aspects of a total space program. Internal niodel analyses, typical of 
the total spectrum available to the user, have been indicated in section 1.2.3. 

Figure 1-1 graphically illustrates selected functions within the model. Considering, the 
major flexibility of the model and its options .previously outlined and discussed in more 
detail in the following sections of the report, this figure and the related discussion 
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should only be considered as representative of the many capabilities and functions which 
are available. Selected comments applicable to this figure follow; 

• Items 1, 2, 3,- and 4 represent input or internal computations derived from 
input. It is notable in 1 that launch vehicle Av performance is computed in 
the model as needed to save storage rather than through the use of "look-up” 
tables. . 

• In items 3 ^d 5 (and elsewhere as appropriate in the model) costs are input 
probabilistically. Item 5 indicates the three basic groups into which cost ele- 
ments are cat^orized for least cost determination in the accelerated search 
subroutine. "H" refers to hardware costs and "L" to launch and other re- 
lated support elements. "Annual" costs are those computed on a yearly 
basis for "sustaining" purposes. Reusable stage programs have investment 
costs associated with them in addition to the conventional three categories. 

Item 3 reflects the special treatment investment costs receive which permits 
analysis pf both reusables and expendables in a total’ program mix. 

• Item 6 is a significantly modified branch-and-bound technique which provides 
an accelerated search of the solution space and outputs the optimal solution 
for the large scale combinatorial problem previously described. 

• Items 7, 8, and 11 indicate types of output available for optimal space pro- 
grams which have not been smoothed under some selected budget ceiling. 

In item 8, @ is tibte optimal assignment while is an alternative sub- 
optimal assignment. Under realistic conditions, however, it is normally 
desired to examine space program options which are both optimized and 
bounded by varying budget limits . When the budget smoothing option is 
selected, item 10 indicates types of constraints which may be imposed. 

• Item 9 represents the wide range of available output options (see section 4. 5) 
and 12 illustrates an example of one of these options. In 12 the smoothing 
option has been selected. 

The foregoing provides a general description of the model, its versatility, and some 
of its solution techniques. Note that the model can be applied to a wide range of space 
program evaluations — to macro -problems that evaluate various options of total space 
programs j to intermediate problems that analyze separate portions of a space program 
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(e.g. , optimizing a scientific, exploratory, service satellite program within a total 
space program), and to micro-problems (e.g. , determining the cost optimal subsys- 
tem among several alternates for a given space vehicle). In all cases the impact of 
cost uncertainty can be assessed quantitatively. 


1-9 



Section 2 

STATISTICAL ANALYSIS OF DATA 


One of the important problems in developing a model whose output is to be used as a 
basis for evaluating tradeoffs concerning future expenditures is the reliability or con- 
fidence in the cost estimates used as input to the model. When applying- estimates to 
specific decisions, the evaluator must be cognizant of the uncertainties that exist in 
the estimated data. Further, to realistically plan programs that can be completed 
within budgetary constraints, he must be able to rapidly and quantitatively assess the 
impact of potentially significant changes in costs in ranking alternatives arid making 
final selection of preferred candidates. 

This section describes an analysis of data sources that apply to advanced systems to 

(1) identify error sources, (2) indicate their relative contributions to total error, and 
(3) indicate statistical characteristics of these error sources. 

2.1 HISTORICAL ANALYSIS OF TRENDS 

To determine the characteristics associated with the uncertainty of predicting costs; 
RAND (Ref. 2) introduces a classification of factors that affect success. These four 
classifications are as follows: 

(1) Costs (development and production) 

(2) Performance 

(3) Time of availability 

(4) Utility 

Performance is used here to include all the qualities of any system that contribute to 
its utility, e.g. , available velocity vs. payload characteristics, maintainability, relia- 
bility, payload capability to a given ephemeris, etc. 
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As used in this classification, utility is largely qualitative in nature, so the future 
utility or benefit of a system is not directly included in the program at this time. The 
analyst may investigate some consequences. of utility changes by performing a sensiti- 
vity analysis on the mission model. The remaining three factors are largely quanti- 
tative in nature and, in practice, tradeoffs among these three are usually possible. 

A given performance normally can be attained earlier if greater costs are incurred, or, 
for given costs, earlier availability is possible if lower performance is accepted, etc. 
The relevance of these tradeoffs to development prediction is obvious — the selected 
parameter may be achieved at the sacrifice of either or both of the other two parameters . 

The Assignment Program assumes initially that performance and availability time are 
fixed. Only, costs are uncertain. While adjusting the rate of actual expenditure to 
accomplish on overall space program within realistic budgetary constraints, the smo- 
othing section of the integrated vehicle assignment and budget; smoothing program may 
change avaUabUity dates, but not performance characteristics'.' Historical data show 
that this approach represents a realistic situation in laimch vehicle programs. Since 
great emphasis is placed on performance requirements, usually cost and/or avaUabUity 
give ground before performance is degraded. Often reliability- is less than expected, 
but the amount by which other performance falls short is usually small in comparison 
to time extensions or cost increases incurred. Since budget constraints as they apply 
to programs continue to be quiet stringent, major emphasis has been given to uncer- 
tainties in cost estimates. 

Uncertainties associated with cost estimates fall into the following three general 
categories: 

(1) Uncertainties due to errors in the costing of the configuration supplied to the 
cost estimator (i. e. , the intrinsic error in cost estimating) 

(2) Uncertainties due to changes in the configuration (e.g. , contract change 
notices (CCNs) as development progresses) {Program Uncertainty) 

■ (3) uncertainties caused by one or more unexpected changes in national economic 
conditions (Economic Uncertainty)- ’ 
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References 3 and 4 detail the five following causes of cost escalation in aircraft and 
missile development: 

(1) System performance, change accounts, for approximately 22% of overruns 

(2) Schedule change ... ~ 6.3% of overruns 

(3) Engineering change in order to meet original performance requirements 
... ~ 10. 6% of overruns 

(4) Economic change ... ~ 15.4% of overruns 

(5) Cost estimate revision .. . ~ 45. 6% of overrims 

Causes (1) through (3) fall into category 2 uncertainties. Cause (3) is primarily due to 
underestimation of the technology advance required. Program uncertainties are di- 
rectly dependent upon the amoimt of technical advance to be incorporated into the new 
system. The ratio of actual total cost to initial estimated total cost varied from 1. 0 
to as much as 17 depending upon the amount of technical advice required in these 
aircraft and missile studies (Refs. 5-9). 

An analysis of spacecraft costs indicates a similar distribution between the five cost . 
escalation t 3 qjes. A detailed breakdown cannot be obtained in all cases because program 
records often list only the net cost change for any one year without specific explanation 
of the cause. In addition to undefined causes of cost escalation, miscellaneous causes 
which do not directly fit into types 1 through 4 are also included in type 5. These two 
factors result in an inflated estimate of importance for this cost estimate revision 
cause. 

t 

As an indication of this cost uncertainty, ratios of actual total cost or late-in-program 
estimates of total cost to initial estimated total cost for various space-related programs 
are provided in Table 2-1. Sources are primarily Refs. 10 to 17, with some additional 
information supplied by program managers. 

The point in the program at which the initial' estimate is taken significantly affects the 
magnitude of factor numbers. Early estimates tend to be extremely optimistic. These 
estimates generally are based upon cost estimating relationships which are historically 
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Table 2-4 


PRELIMINARY COST RATIOS 


Program 


Reference 



Ratio 

Reference 

SRAM 

3.49 . 

14. 


SIVB <SV) 

5.49 

16. 

Titan 3-C Dev. 

1.59 

10 , 


CSM 

1.29 

16 

Prod. 

1.00 

10 ' ■ 

• , 

sn 

9.88 

16 

Pershing I Dev. ■ 

1.62 

10^ . 


SIVB (SIB) 

1.73 

16. 

Prod. 

0.94 

10 


Mercury S/C 

7.3 

16 

LOH Dev. 

2.25 

10 


Gemini 

2.71 

16 

Prod. 

0.89 

10 


Viki% 

2.28- 

14 

XC-142Dev. . 

1.79 

10 


ERTS A-B 

4.35 

14 

F-111 Dev. 

3.93 

10 


ATS 

1.44 

16 & 17 

Prod. 

1.92 

10 


Intelsat I<4n 

2.25 

14 

Program 

2.23 

14 


GEOS A-B 

1.38 

14 

C-141 Dev. 

1.38 

10 . 


Nlmboa A-D 

1.27 

14 

Prod. 

1.51 

10 


Tiros -M 

1.34 

14 

LANCE Dev. 

3.76 

14 & 10 


Lunar Orb iter 

1.72 

14 

SPRINT 

1.26 

10 


Pioneer A-E 

1.97 

16 & 17 

Cheyenne 

2.86 

10 . 


Surveyor 

5.41 

. ' 14 

C-5A Dev. 

1.14 

10 . 


, OAO A-C 

2.92 ^ 

16 & 17 

, Prod. ■ 

. 1.60 

10 


OGO 

1.63- 

16 & 17 

Total 

1.47 

14 


OSOA-H 

1.90 

16 & 17 

Centaur 

8.29 

16 


RAE A-B 

2.79 

14 

Scout 

4.43 

16 


ApoUo S/C 

3.14 

16 

Delta 

1.97 

16 


Ranger 

. 2.29 

JPL i 

LEM 

5.55 

16 


Mariner 64-67 

2.00 

JPL 

SIB 

2.12 

16 


' Mariner 69 

1.31 

JPL 

SIC 

2.30 

16 

















derived, and may cover less than Is later understood to be essential. They generally 
understate the technological difficulty involved in a given enterprise and the cost of 
many indirect contributors to total program costs - or even to development costs. 

The importance of the time of initial estimate is shown by Fig. 2-1, which is presented 
in Ref. 12 from unpublished data collected for the Marshall-Meckling study. The curve 
plots cost factor numbers for a group of fighter aircraft developed in the 1950*s against 
the time at which the initial estimate was made. The horizontal axis is measured in 
months before Initial Operating Capability (IOC). The zone designated A is roughly 
representative of time at which a Technical Development Plan for fighter aircraft 
probably wotdd be approved today. Zone B, somewhat higher on the curve, is probably 
representative of the period during which a definitive contract emerges or a firm con- 
tract target is established. The significant point, of course, is that if observations 
are taken earlier or later than at A or B, quite different factor numbers will result. 

As shown by the points on the curve, the observations plotted for programs of the 1950*6 
differ widely in their distance from IOC . 

The curve itself, although representative of only one lot of fighter aircraft programs, 

is strikingly like estimating relationship curves derived by Summers (Ref. 2) for other 

kinds of aircraft programs and missile developments during the 1950*s. Because the 

object of the survey presented in Ref. 12^was to examine the ability of the Services to 

predict and control program outcomes in the 1960's in comparison to the 1950's, the 

* 'y* 

bias introduced by taking initial estimates at no definite time in the program was 
avoided by selecting estimates in Region A whenever available. Region B estimates 
were used if none were available for Region A. The conclusion reached was that, on 
the average, estimates in the 1960’s were 25% less optimistic than in the 1950's. Thus, 
Fig. 2-1 should only be used to indicate the relative (not the absoliU^i^hange in^^t 
factor as the time of initial estimate changes. The cost factor differential will not be 
as great for 1960 estimates. 

However, the advanced planner who is interested in performing tradeoffs between pro- 
posed programs does not have estimates In Regions A or B available for use. He must 
rely on early estimates available for his analysis. Therefore, whenever possible, 
these initial estimates were used as the reference for the ratio presented in Table 2-1. 
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Fig. 2-1 Cost Factors 
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/ final cost or last estimate of final cost\ „ . . ' 

(ratio = plamilDg estlmate ~~ J ' change m program, soope oc- 

eurred between the initial and final estimates, the costs were adjusted accordingly, A 
statistical an^ysis based on the ratios presented in Table 2-1 cannot be performed at 
this time because several of the programs involved were not defined precisely enough 
at the time of the initial estimate, so that it cannot be determined if adjustments to the 
data are required or not. More information on these programs is required. Another 
problem involves the availability of initial estimates. Eatios based on information in 
Ref. 10 have initial- estimates in Region A or B. Estimates in Region C were not avail- 
able for these programs. Until these biases can be removed, the ratios may only be 
used as guidelines for the analyst when estimating the uncertainty of available program 
estimates. 

Two characteristics of program costs are evident from the analysis performed to date. 
Spacecraft programs often have a ratio close to unity. R.such a program is in danger 
of significant overruns, then the number of laimches is frequently cut or the program is 
reduced in scope, resulting in either small total program overruns or actual underruns. 
Cost adjustments compensate for this characteristic, provided the original program 
was defined clearly. 

Many of the overruns could be accounted for by an increase in final inert weight over 
that initially proposed. Spacecraft costs tend to increase linearly with weight increases, 
while launch vehicles or other propulsive systems- esiperience an exponential growth in 
cost due to increased propellant requirements, weight growth in structure, and other 
subsystems due to the initial weight growth increment. For example, weight growth of 
25:1 is not unusual as a ratio of resulting increase in liftoff weight to an increase in 
orbiter weight for an earth orbital vehicle. 

Launch vehicle programs normally include recurring costs which may vary considerably 
over long periods of time from that originally planned. Ratios comparing RDT&E costs 
and recurring costs individually are more informative than just a comparison of total. • 
program costs; however, the allocation of funds between RDT&E and the production 
item is not always clear-cut. When available, the separate ratios are presented. 


2-7 



Some programs are partially funded by other programs. If the degree of sponsorship 
can be ascertained, the costs were delegated to the appropriate program. Otherwise, 
(see Mariner 64 and Mariner 67) the program costs were combined before ratios were 
taken. 

Figure 2-2 indicates the shape of the preliminary cost factor distribution taken from 
Table 2-1. Only cost factors which could be reliably unbiased were used in this figure. 
Both propulsive and non-propul sive entries were combined in order to smooth the re- 
sults. However, propulsive and nonpropulsive cost factors have the same general dis- 
tribution when plotted independently. The main difference is the tail is longer on the 
propulsive distribution as discussed above. 

In conclusion, the cost growth ratios presented in Table 2-1 were based on historical 
data currently available. More work needs to be done before a complete statistical 
analysis can be based on them. They may be used as guidelines for the analyst when 
estimating the uncertainties associated with future program cost estimates. 

2.2 COST ESTIMATING UNCERTAINTY 

The uncertainty involved in cost estimating errors refers to variations in program 
costs when program and economic uncertainties are zero, i.e. , when the vehicle con- 
figuration and all other program and economic parameters are held constant. Reference 
5 lists the major sources of cost estimating errors as: 

• Cost Estimation Relationship (CER) Errors 

• Data Errors 

• Extrapolation Errors 

• Aggregation Errors 
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Fig. 2-2 Cost-Factor Distribution . 


Numerous other sources of uncertainty — such as bias in the analysis, differences in 
interpretation of hardware requirements, omission of elements' in defining an overall 
program, which for some technology may also be unavailable (i.e. , "unknown unknowns," 
Ref. 18) - are possible for ekplaining the total variations in cost estimation. However, 
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the foregoing four error sources constitute a major part of the problem and are dis- 
cussed below. 

CER errors are expected because cost estimating relationships can be assumed to hold 
only within certain tolerance limits. The cost and engineering data used in derivii^ 
CERs invariably contain errors also, due to the complexity and vast quantity of data 
involved. Extrapolation beyond the range of the samples or data base from which the 
CER was derived is frequently required since the cost analyst does not have another 
alternative at his disposal. Extension of the CER into a new region can introduce sig- 
nificant errors. These circumstances indicate some of the potential sources of error 
in applying CERs to estimate the cost of new systems. 

Aggregation errors arise because each analyst sums the cost elements differently. 
Possibly, the sum of these errors is negligible, due to counterbalancing between indi- 
vidual errors; however, this possibiliiy cannot be assumed in general. Although Refs. 

2 to 16 and 19 to 22 provided valuable background material for this study. Refs. 23, 24, 
and 25 were especially useful because they explore in detail the statistical characteris- 
tics associated with cost estimating errors associated with launch vehicle programs. 
The conclusions in general are as follows: 

• Sample standard deviations range from 31 to 90% of corresponding mean value'. 

0 Nature of each sample distribution indicated it was unimodal and either nearly 
S 3 rmmetric or skewed positively. Total program cost was unimodal and either 
symmetric or skewed positively. (Reference 25 indicates that RDT & E cost 
uncertainties tend to be positively skewed while investment and operations cost 
imcertainty are more likely to be symmetric in their distribution. ) 

0 Since existing technology was assumed in order to minimize program uncer- 
tainties, RDT&E program costs were relatively certain. Under these circum- 
stances, operational program costs showed the -greatest variation due to large 
discrepancies in estimating imit flight hardware cost. 3h advanced technology 
programs, RDT&E costs will also have significant cost uncertainty. 
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Category 1 uncertainties are as large as the program {or Category 2) uncer- 
tainties discussed in section 2.1, If an extensive development program had 
been postulated for this analysis, the variation of the RDT&E estimates would 
have been much larger .and further increased the variation in total program 
costs. 

® Although lower-level cost uncertainties are skewed, their independent sum 
quickly converges to the normal distribution. Thus, the observed skewness 
of the aggregate program cost uncertainty has to be due to one or more of the 
following possibilities: 

(1) One cost element dominates all others in magnitude. 

(2) Program elements are correlated. 

(3) Uncertainties (other than cost estimating uncertainties) such as engine- 
ering- change, schedule change, and system performance change imcer- 
tainties are important. 

2’. 3 PROGRAM SCHEDULE COST RELATIONSHIPS 

Accelerated and stretched-out development programs result in increased expected costs 
over the nominal program development time. The relationship between time changes in 
the development program period and increased costs was determined for a typical pro- 
gram having an 8 year nominal development period and involving both a spacecraft and 
launch vehicle development. Refs. 2, 16 {1965 in partictilar), 26, 27, and 28 were 
principal sources for this analysis. 

The determination of the relationship between development costs and phasing of develop- 
ment begins with the cost analysis illustrated on Fig. 2-3. The direct costs consist-of 
manpower and material. Manpower can be broa(Ry separated into manufacturing labor 
and engineering labor. Material consists of purchased raw material/supplies, minor 
subcontract items, and major subcontractor costs. The indirect costs are those re- 
quired to support directly the engineering, manufacturing, planning, and material- 
processing efforts, and the overhead and general administrative costs which are re- 
quired for maintaining the physical plant and providing administrative supervision, 
clerical support, and related effort. 
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Fig. 2-3 Cost Analysis 



Fig. 2-4 Spacecraft Program Phasing — Design 



The groundrules for this analysis, which has been extracted from NASA reports, include 

• Nominal development plan and program elements are the base for the study. 

• Delayed schedules are to assume a stretchout of nominal program elements 
without major revision to the definition of these elements. 

• The schedules analyzed include: 

N “ nominal schedule 
A — 6% acceleration 
B — 37. 5% delay 
C — 75% delay 

Figures 2-4 through 2-6 illustrate how the spacecraft program phasing varies with 
schedule. These charts divide the total project activity into periods of design, manu- 
facturing, and test and compare the four different schedules above in terms of period 
of time occupied in each of these activities. Such phasing analyses are essential in 
arriving at costs associated with each category of effort. Figure. 2-4 divides the sched- 
ule for spacecraft design into two parts, that for basic design of the components and 
that of the engineering support required to carry on the design changes for each of the 
schedules. As a complicated research and development program develops, many design 
changes become necessary as development data feed back from ground and flight tests. 
As .the flight program is stretched out, then the design period must also be stretched. 
For the longest stretchout considered, the design period for incorporating changes 
would actually stretch out to be almost three times as long as for the nominal schedule. 
Therefore, a minimum staff of design engineers must be maintained to accommodate 
these changes for that period of time. Since the design work requires many different 
disciplines, a minimum level of capability is needed in each of these different areas. 
Thus a minimum engineering staff is required for the ability to respond to all of the 
changes that may be necessary as the flight program proceeds. The size of this staff 
is relatively independent of the number of launches attempted each year. If this staff is 
not fully utilized because the program was stretched, then money is being spent 
ineffectively. 
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Fig. 2-5 Spacecraft Program Phasing — Manufacturing 



Fig. 2-6 Spacecraft Program Phasing — Test 
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In manufacturing, boiler plates, prototype spacecraft, and production or operational 
spacecraft must be built. The spacecraft prototype may be delayed somewhat, but de- 
lay beyond a certain point results in large increases in spending because the prototype 
fails to serve its intended purpose. 

The main stretchout in activity would be in the fabrication of production spacecraft. 

The direct production effort is more suitable for efficient stretchout than is the direct 
engineering effort . The direct manhours required for manufacture of production space- 
craft are close to constant regardless of the schedule (Ref. 26). Overhead increases 
significantly, however, for the stretched out manufacturing- period. 

There are four major phases in the test activity: (1) subsystems development, (2) sub- 
systems qualification, (3) systems ground tests, and (4) systems flight tests. Phases 
(l) and (2) are stretched out in accordance with the first flight date to which these 
phases are tied. Phases (3) and (4) have significant stretchouts for the different flight 
schedules under consideration. 

In estimating the prime contractor costs, a step-by-step an^ysis of the m anh our re- 
quirements for engineering and manufacturing tasks that must be performed is first 
made. Then the subcontractor items are similarly analyzed with assistance from 
major subcontractors involved. 

Once the direct costs have been estimated, the management and overhead costs re- 
quired to support the direct effort are added. The amount of overhead does not de- 
crease in proportion to a decrease in direct labor. Keeping a facility open and running 
costs some minimum amoimt, whether any directed effort is applied or not. The over- 
head charged also varies with the total amount of company business at a given time. 

The relative overall cost estimates for each major component's development under the 
four schedules considered are presented in Figs. 2-7 through 2-9. 
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These schedules are combined and analyzed to produce an overall program cost for 
each component and then for the total program. Since there is more than one way to 
combine component schedules in order to arrive at total program cost for a fixed 
completion date, there is a range of total costs which can be expected. A reasonable 
range is indicated on Fig. 2-10. Figure 2-11 summarizes total costs for the optima] 
combination of components for each program completion time considered. The 6% 
accelerated program results in at least a 6% increase in total cost. The 37.5% de- 
layed program increases costs.. 15% while the 75% delay causes a 35% increase in 
total program cost. .An analysis of how the cost varies by year within the program 



F^. 2-9 Engine R.&D Cost Variation 
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Fig. 2-11 Program R&D Cost Summary 
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shows that a delayed program does result in some cost reduction for certain years. 
However, the sum over all the years is greater than the nominal case. Since the 
computer model developed distinguishes between launch vehicle development pro- 
grams and spacecraft development, Figs. 2-7 and 2-8 are of special interest. 

The preceding analysis indicates that certain fixed operating, personnel, and facilities 
costs account primarily for the increase in total costs of a decelerated (stretched) pro- 
gram. For example, a significant number of skilled engineers, scientists, and techni- 
cians are needed to support the flight and ground-test activity that is required through- 
out the development program. These personnel include propulsion, electronics, struc- 
tures, thermodynamics, astrodynamics, and guidance and control specialists, as well 
as supporting technicians, plus the clerical and management staffs required by each 
industrial contractor to support the effort. 

All^hese costs remain practically the same on a time basis regardless of program pace 
or launch rate'. Because these costs accumulate in almost direct proportion to the time 
required for program completion, a stretchout would substantially increase them. It 
would also reduce useful output and require maintenance of a technical -industrial base 
in low -gear operation over a longer period of time. 

An accelerated program, on the other hand, reduces these fixed costs since development 
time is shortened. However, parallel efforts are often conducted in this situation on 
essential subsystems/components that have high technical, risk to increase the probability 
of satisfactory completion within the shortened period of time. These parallel efforts 
increase costs substantially. Even if cost were not a limiting factor, no more than a 
30% acceleration is allowed in general due to technological limitations (Refs. 26 and 27). 
These relationships and limitations are part of the smoothing portion of the existing 
model. . 
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2,4 CONCLUSIONS 


As a result of this preliminary survey of historical data, the following conclusions are 
apparent: 

(1) Cost estimates of important parameters made early in a program are usually 
quite inaccurate in two respects. First, these estimates are strongly 'TDiased" 
toward overoptimism. Second, aside from bias, errors in estimates evidence 
a substantial variation. 

(2) The accuracy of estimates is a function of the stage of development; i. e. , 
estimates improve as development of the item progresses. Similarly, esti- 
mates for development projects representing only "modest advances" tend 
to be better than for more ambitious projects. Further, the actual length in 
time of a development program directly influences the variation in cost 
estimate. 

(3) > Cost estimating xmcertainties behave statistically as though they were dis- 

tributed on a unimodal curve which is either symmetric or .(most often) skewed 
left. The parameters associated with this curve depend on the type of program 
being estimated and exhibit characteristics as described in section 2.2. 

(4) Mission (payload) related cost uncertainties have the same general charac- 
teristics as launch vehicle costs; however, the magnitude of expected mission 
costs is greater while the imcertainty is less than for general launch vehicle 
costs. 

- (5) Even though independent sums of skewed distributions quickly converge to the 
normal distribution, the mean (and mode) of the approximating normal dis- 
tribution is not the sum of the modes of the component distributions. Thus 
, merely summing the most lilcely costs of component programs does not give 
a valid indication of the cost of the total program. 

(6) As program development time is increased (development stretch), costs in- 
crease mainly due to certain fixed costs being spent over a longer period of 
time. As program development time decreases (development acceleration), 
parallel efforts often take place, causing an exponential Increase in total costs. 
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Section 3 

ANALYTICAL APPROACH 


The development of a model which compares risks between different programs and 
estimates the probability of program costs exceeding cost estimates must be based 
upon historical data. This section compares various statistical distributions in terms 
of their ability to meet the requirements determined in Section 2. The log-normal 
distribution is selected as most useful. The characteristics of this distribution are 
provided as well as a description of the analjd;ic* relationships incorporated into the 
model which transform the model from a deterministic to a statistical evaluation tool. 

3.1 DATA DISTRIBUTIONS 

The statistical analysis of cost data .presented in Section 2 has indicated the type of 
distribution which best describes the anticipated cost uncertainties. Figure 3-1 
indicates the general shape to be expected. 



MEDIAN 

MODE MEAN 
^ST ESTIMATE 

Fig. 3-1 Distribution of Cost Estimate- 
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There are three general distributions which have the characteristics shown in Fig. 3-1; 

(1) the log-normal (A) distribution, (2) the gamma (F) distribution and (3) the beta (p) 

distribution. The T distribution has the real axis from 0 to infinity as its domain and 

2 

for some values of the parameters looks like the distribution in Fig. 3-1. The x 

distribution (a special case of the P distribution) has the added property that the sum 
2 2 

of any number of x distributed variables is again X distributed. There is one major 
disadvantage to the use of this distribution, however. Although the P distribution has 
two parameters, one parameter is only used for ''sizing" and, hence, many shapes 
anticipated for actual error distributions cannot be approximated accurately. 

The /3 distribution (and the triangular distribution sometimes used to approximate the 
beta) is a "real" two-parameter function allowing close approximation to all the antici- 
pated distributions including the symmetric ones. Both positive and negative skewing 
may be approximated. The main disadvantage of this distribution is ;that,the unit inter- 
val is its domain and thus normalization is required. Since there is a positive probability 
that costs may exceed any given number, normalization is an arbitrary .procedureTimit- 
ing.the usefulness of this distribution. 

The log-normal distribution has the advantage of being both a "real" two -parameter 
function, and a function whose domain is the positive real axis. Thus, close approxima- 
tions are available for all positively skewed and symmetric distributions. Since there ' 
is a positive probability of reaching any positive real number, normalization is not 
necessary. The sum of many variables, each distributed log normally, is not neces- 
sarily distributed as another log normal function. However, historical data indicate 
that the total cost should also be distributed as in Fig. 3-1. Therefore, key parameters 
describing the sum are used to approximate the sum as another log normal variable. 

Other distributions, similar to the three discussed above, were found unsuitable for 
the analysis for some of the same -reasons given above. 

Because the log-normal distribution is easy to work with and has the essential character- 
istics dictated by historical data, it was chosen as being most representative of the 
expected cost-error distributions. 
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3.2 CHARACTERISTICS OF THE LOG-NORMAL DISTRIBUTION 


While the exponential distribution makes use of the arithmetic mean of the variable, 
the log-normal distribution makes use of the geometric mean of the variable, or the 
arithmetic mean of the logarithm of the variable. Since the log-normal distribution is 
basically a two -parameter distribution, it cannot be specified by the mean alone. 

The log-normal distribution is skewed, but approaches the normal as the standard 
deviation of the associated normal distribution approaches zero. It has been character- 
ized by some as a model of the law of proportionate effect and has been shown to be 
applicable to many economic, biologic, and advanced technologic processes (Refs. 29 
and 30). 

A random variable is said to have a logarithmic normal (log-normal) distribution if 
the logarithm of the variable is distributed normally. If x is a positive variate . 

2 

(0 < X < «>) and if y = In x is normally distributed with mean fi and variance a , 
then X is said to be lognormally distributed. The distribution function may be written 
as 


f(x) = 


X(T 


V2^ 


_ (lux - M)" 
2u^ 


(3.1) 


where 

{j, = In X 

0 -^ = var (In x) 

The following relations hold: 

median of f (x) = e^ 

t 

mean of f (x) = ^ ^ 

2 

mean of f (x) = 


(3.2) 
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Thus, one can calculate values for f(x) using standardized normal tables for 
f (y = In x). 

3. 3 INTEKNAL PROGEAM MODIFICATIONS 


Input is provided basically by two costs for each item instead of the single cost used 
in the deterministic model. The most likely cost, m , is estimated first, based on 
the most realistic estimate available. Next a pessimistic cost, b, is estimated such 
that the probability of exceeding this cost is x% where x is less than 50. The choice 
of m and b will determine how skewed the distribution will be. The nature of the 
item must be taken into consideration when estimating m and b as discussed in 
Section 2. Maximum cost estimates should include the possibility of initial failure 
and a new start in one or more components of a program or of program redefinition. 

Using the relationships (3.2) presented in Section 3. 2, the input data are developed 
as follows: 

For each cost, two values are input: 

_ 2 

m = mode =■ ^ 

XX = x% tail such that prob (Y > xx) 


= x/100 


(3.3) 


Y is defined by N(Y j O , 1) = 1 - x/lOO, N being the normal cumulative distribution, 
so we have 


a (parameter) 


-Y + Y + 4ln 




1/2 


3/2 (X 


E (mean) = m e' 


(3.4) 



and 


2 (T^ 

^ (variance) = E (e - 1) 

E and a are stored for each cost since all other variates are functions of these two. 
The algorithm proceeds as before to find a solution based now on expected values for 
all component costs. Once this solution is found, the appropriate statistical parameters 
are used to determine the distribution characteristics associated with the total program 
cost for this solution. The individual expected values are simply added to determine 
the total program expected cost [Eq. (3. 6)] . The variance is found assuming either 
complete independence or complete dependence between variables depending upon the 
situation, [Eq. (3.7)]. For example, recurring costs for stage X in year Y are 
assumed to be completely dependent on these same costs for year Y + 1. The algorithm 
continues to find solutions, whose total expected costs are placed in ascending order, 
until n solutions have been found where n = NSOL is an input variable. As each 
solution is found, the corresponding assignment is printed out along with information 
concerning its total cost distribution and its relation to other solutions found previously. 

If inflation at an average rate p = GRO is input, then the relationship used is 

N 

m' (in year Y + N) = (1 + p) m (in year Y) 

2N 

var' (in year Y + N) = (1 + p) var (in year Y) 

Parameters for the total program cost are calculated using Eqs. (3. 6) and (3. 7). 

kg (1 + p) + k3 (1 + p)2 + . . . + kjj (1 + p)^ 

i i i 

(3.6) 


E (total cost) 
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var (total cost) 


(1 + p) + kg (1 + p) + . . . 


1 2 2 

+ (1 + p)^"^ {e*^! - 1) (3. 7) 

i J 


where 


k. = number of times cost i is used in year j, 1 < j < N 
'^i 


The effect of dependence (correlation) on uncertainties associated with input data may 
be displayed graphically as in Fig. 3-2. For each variable associated with one program 
element (the two chosen here are the development cost and the operating cost of one 
reusable vehicle), the parameters and cr are calculated for the associated normal 
distribution. Then 


/T ' 2, 

(1 - P ) 


- Px) , (y - 

2 ~ cr or 2 

cr X y .or - 

X y -I 


(3.8) 


determines the probability ellipse in the normal plane. 


If we assume that o-^ = cr^ (i. e. , the two variables have the same growth factor), then 

the major axis = cr |'G(1 + p)]^^^ and the minor axis = cr [G(l - p)]^^^ with the center 

at (x , y). This ellipse in the normal plane is then mapped back into the log-normal 

plane, yielding results similar to those shown in Fig. 3-2. For a 50% probability 

2 

ellipse G = (1. 177) . Since the area of an ellipse is xab. Table 3-1 shows, under 
the above assumptions, how the area of a fixed probability ellipse decreases as the 
amount of correlation increases. 
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Table 3-1 


ABEA OF PEOBABILITY ELLIPSE 


Correlation 

(P) 

Major Axis 
(a) 

Minor Axis 
(b) 

Area 

0 

1. 177<J 

1. 1770- 

ircr^G 

0.1 

1. 295o- 

I.O 60 - 

0. 995 -na^G 

0.25 

1.4630- 

0. 8840- 

0. 965 na^G 

0.4 

1.650- 

0.701a 

0. 9165 ira^G 

0.5 

1.770- 

0. 59o 

0 . 866 Tta^G 

0.6 

1. 89o- 

0.472cr 

. 0 . 80 Tta^G 

0.9 

2.24o- 

0. 117(7 

0. 436 ira^G 

1.0 

2. 34o- 

0 

0 


Because the areas described in Table 3-1 have complex shapes in the log-normal plane, 
more sophisticated equipment than the impact printer is required to produce them. 

Since the computer program is to be as versatile as possible, in the present work the 
above results- are not produced directly but may be derived from the data output. In 
the more complete case, with three basic cost categories input, the uncertainty region 
for a given probability will be a volume having log-normal characteristics. A CRT 
display, which has better capability for presenting three dimensional data, would pro- 
vide quicker reaction and more lucid output results. 

In the present model subroutine SMOTHS plots total costs on a yearly basis with their 
corresponding statistical parameters. Payload costs are treated in the same manner 
as launch vehicle costs. If a development program is stretched or accelerated in this 
subroutine, the expected development cost will increase, as explained in Section 2. 
Based on historical data, the uncertainty associated with a stretched program will 
also increase. However, although introduction of parallel development approaches in 
some of the component developments of an accelerated program will increase the 
expected cost of the development, a reduction in the dispersion or risk of the outcome 




and possibly a reduction in the uncertainty associated with the new expected cost may 
result (Ref. 16). Thus new expected costs and uncertainties are calculated in SMOTHS 
whenever a program development period is changed. 


3.4 OUTPUT 


The analyst will be attempting to select a fleet of launch vehicles and associated pro.- 
gram elements to accomplish a proposed set of missions from alternative combinations. 
He will want to determine the margin of cost difference between alternative choices. 

A wide variety of output is' available from the algorithm since, for each solution, the 
log-normal distribution with its associated parameters is known. Equations (3. 6) and 
(3. 7) define the expected value and the variance of each total program cost. The 
parameters, cr, /i, for each such assignment may then be found using 


var (TO ^ 
[E (TC)f 


(3.9) 


and 


2 

M = In [E (TC)] - — . 

The most likely value, m , for each assignment is determined by 


m 


mode = E (TC) 



(3. 10) 


The probability that the total cost will not exceed some value Y may be found from 
the following relationships; 


prob (X < Y) = p which is equivalent to N (Z | 0 , 1) = p (3. 11) 
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where 


Y = 


The scientific subroutines NDTBI and NDTR can be used to find Z given p or p 
given Y, respectively. Using the above relationships, the probability that the expected 
program value (mean) will exceed the estimated value (mode) is determined. 

To compare two assignments, the probability that one assignment will actually cost 
more than the other should be known. The log-normal distribution allows such a 
determination providing that the degree of correlation between programs is provided. 

Thus, two assignments involving different development programs may be highly corre- 
lated if each development program involves the same type of risk, or they may be only 
slightly correlated if one involves a large hew development and the other utilizes exist- ' 
ing technology to accornplish the same mission profile. 

Two assignments with total costs and distributed log-normally will have 
parameters [V (TC^), E (TC^)] and [V(TCg), E(TCg)] determined by Eqs. (3. 6) 
and (3.7). The parameters (p^, a^) and o-^) may be determined by Eq. (3.9). 
Then log = log - log.C . - is normally distributed with mean = p - p . and 

variance = cr^ + - 2p ctj^ cr^ where p is the correlation coefficient, discussed 

above, which describes the relationship between assignments C. and C„.- 

Thus the probability that assignment B will cost less than assignment A is 




h 


B 


= k and p given j = Pr Hn ^ < 0 j = N ^.0 I 


mean 


2 2 \ 
and cr ^ ~ variance) 


= N 


Ma - P 


B 


(“■a + 4 ■ ^p<^a<’b) 


> 1/2 


A'^B 

0 , 1 


for p < 1 
(3. 12) 
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The probability expressed in Eq. (3. 12) is output for representative values of p for 
all pairs of assignments of interest so the analyst may obtain insight into the inter- 
relationships between the assignments. Possible comparisons include the following 
two examples: 



E (B) > E (A) but Prob (A > B) is 
large. Choice depends on outside 
factors. (A and B represent alter- 
native total program assignments . ) 


B 



MODE (B) > MODE (A) so deter- 
ministic analysis would choose 
assignment A; but E (B) < E (A), 
Therefore, using estimated costs 
(MODE) alone results in invalid 
conclusion. Statistical presentation 
shows complex relationships so that 
valid conclusions can be determined. 


The optimal assignment is displayed in SMOTHS with the results printed year by year. 
The graphical display includes the 50 percent confidence limits for each year (Fig, 3-3), 
Additional data on output modes are provided in Section 4. 5 and Appendix B (Vol. 2). 
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Section 4 

COMPUTER MODEL DESCRIPTION AND OPERATION . 


This section describes the probabilistic optimal assignment and budget smoothing 
model developed during this phase of work. The logic for the model is described in 
this section and is detailed in Appendix C (Vol. 2). A summary description of variables 
which may be input to the program is included in Table 4-1 (presented in Section 4. 2). 
Appendix A (Vol. 2) lists the input requirements in detail along with a glossary of input 
terms. A sample case illustrating the type of probabilistic input and .output which may 
be generated by this model is included in Appendix B (Vol. 2). The sample case may 
also be used for program checkout. This section indicates the flexibility of the model 
available through, its many options. Also included are examples of sensitivity tradeoffs 
that can be derived from output runs using the model, and the application of the model 
to advanced technology systems. 

4. 1 LOGIC 

The optimum assignment program is integrated with the budget smoothing program 
through use of a master program which translates from one model to the other. The 
deterministic budget smoothing program whs developed by R. E. Slye, the Technical 
Monitor for the study, and has been described in Ref. 1. This smoothing program 
was extended to handle probabilistic input-arid to output budget levels showing inherent 
cost uncertainties. It is therefore further discussed in this volume. A general logic 
diagram of the master program and the two main subroutines, ASIGNS and SMOTHS, 
are presented in Figs. 4-1 through' 4-3. 

The master program (MASTER) calls first the vehicle assignment program (ASIGNS) 
in order to obtain mission data, cost data, and optimima vehicle-to-mission assignment 
based on these data. Input data are output using both modal and expected values if 
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CALL 

ASIGNS 


MYRS^ 


0 < MYRS < 21 


END OF 
DATA 


TERMINATE 

•RUN 


CALCULATE VARIABLES 
FOR ASIGNS- 


OPTIMUM VEHICLE/MISSION ASSIGNMENT HAS 
BEEN DETERMINED 


AT LEAST ONE SIGNIFICANT 
VARIABLE TO ASIGNS WAS 
CHANGED IN SMOTHS SO A 
NEW ASSIGNMENT MAY BE' 
POSSIBLE 


CALCULATE VARIABLES FOR SMOTHS 


CALL SMOTHS 


FINISH <MITR-H 


FINISH 


SMOTHS CHANGED NO 
SIGNIFICANT INPUT 
VARIABLES TO ASIGNS ' 
HAVE OPTIMUM 
SMOOTHED PROGRAM 


MITR+ 1 


JFLAG = 


£^FLACT 


MAXIMUM NUMBER OF 
ITERATIONS EXCEEDED 


JFLAG 


Fig. 4-1 General Flow Diagram for MASTER Program 


4-2 







4-3 



Fig.. 4-2 General Flow Diagram for ASIGNS Program 














































appropriate. The N best solutions based on expected costs are output along with 
their statistical relationships, but only the optimal assignment is saved for .use by 
MASTER. MASTER then transforms this data from the optimal assignment so that 
it may be used directly by the budget smoothing program (SMOTHS). SMOTHS shifts 
development dates, launch dates and development duration to achieve a level of spending 
close to the desired level. The desired levels of spending and constraints on possible 
program shifts are input to SMOTHS directly. Annual spending levels are output by 
SMOTHS based on expected costs and most likely costs. A 50% confidence interval 
about the expected cost is output and displayed on each plot of annual spending levels. 

The new development dates and development costs generated by SMOTHS are trans- 
formed by MASTER so that ASIGNS can use the data for a revised vehicle to mission 
assignment. The program iterates between ASIGNS and SMOTHS until no major changes 
are generated by SMOTHS, Then MASTER either terminates ,or starts a new case with 
associated data. 

Figure 4-4 illustrates the overall relationship between the 32 subroutines. Subroutines 
INPUT and PLOT are available to all NASA computer users and are described in 
Appendix C. ' Subroutines PACK and AFRMT were written in 360 Assembler Language 
by R. E. Slye, the Technical Monitor of this study. Listings for each are included in 
Appendix D and. a description of both subroutines appear in Appendix C. The remain- 
ing subroutines have flow charts in detail in Appendix C and Fortran listings in Appendix 
D. The first comment card in each subroutine listing states the primary purpose of 
that subroutine. Other comment cards describing the purpose of each section and 
defining any pertinent variable whose name- is not mnemonic are distributed liberally 
throughput the listing so that new users may familiarize themselves with the logical 
function of each subsection within the program. 

Dimension restrictions are detailed in Appendix A for input variables and for internal 
variables indirectly associated with the input. All other dimension constraints, data 
statements, and equivalence relations may be found at the beginning of the program 
listing for MASTER in Appendix D. 
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Fig. 4-4 Program Subroutine Relationships 
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Each subroutine has been constructed as a self-contained package with a minimum 
of interrelationship between routines. Consequently, any subroutine can be altered, 
expanded, or modified with the minimum amount of effort. The length of each sub- 
routine was restricted so that maximum use of the Fortran H mode of compilation 
would result. This efficient mode of compilation results in reduced storage and reduced 
run times in comparison with the more common Fortran G mode. 

4. 2’ GENERAL INPUT REQUIREMENTS 

Detailed input requirements are included in Appendix A (Vol. 2). In general, cost data 
may be input as a most likely value (modal value) for each type, plus an xx% tail value 
for each t 5 q)e, where xx is an input value. If xx is input as zero for any cost, then 
the model interprets that cost as being certain so the modal value for that cost equals 
the expected value. If xx is zero for all cost data input (i. e. , there are no upper tail 
values given), then the program bypasses all statistical calculations. In this manner 
the deterministic model reported in Ref. 1 was embedded into the present probabilistic 
model. 

As in the deterministic model, input cost data may be related to individual stages, a 
family of stages, interstage integration, or launch facility. At present a launch facility 
is either ETR or WTR while a launch complex consists of at most three specific pads, 
at one of these facilities. Unit or recurring costs are expended at each launch. Fixed 
or one-time costs may be spread-out over a period of years, but are only spent once • 
and must be spent before the component is considered operational. Annual costs include 
all sustaining “type costs, and represent any cost computed on a yearly basis. All input 
costs are grouped into one of these three categories, but may be related to any of the 
sources listed or to a particular vehicle if more convenient. 

An outline of the input data which may be included in the program is shown in Table 
4-1. In contrast to Appendix A, input data listed in this table is provided for the 
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Table 4-1 


INPUT DATA 

VEHICLE BELATED INFOBMATION 

A. Costs 

1. Stage Name and Other Belated Data 

Eecurring (hardware and launch, site (ETB or WTB) operations) — 
first unit and learning curve percent. 

Development — including the year in which development begins and 
the number of years over which development cost is distributed. 

Sustaining — including stage sustaining at launch site. 

2. Shared Cost Data — Name of group, sustaining- and development 
including the year in which development starts and the number of 
years over which the development cost is distributed. 

3. Integration Cost Data (between two shared groups) 

Development — including the number of years over which the develop- 
ment cost is distributed and the year in which development starts. 

4. Pad/Facility Costs — development and sustaining costs for stage, 
family, and integration. (May be input in terms of individual pads 
or in terms of a facility. ) 

5. Investment Cost for Eeusable Stages 

Number of units for initial investment (May be input or program 
determined from the launch schedule and other reusable input data). 

First unit price. 

Amortization lifetime in terms of number of launches (optional). 
Turnaround time in days and learning curve percent (optional). 

6. Miscellaneous Costs — Any costs which do not fit in the above categories 

B. Performance 

1. Name of stages which constitute vehicle 

2. Payload vs. velocity curve constants 

3. Stabilization requirement 

4. Manrating requirement 



Table 4-1 (Cont. ) 


5. Number of restarts required 

6. Pad complex constraints 

7. Return payload weight of reusable upper stage (input in stage section) 

C. Availability 

1. Final year in which the stage is available (Initial availability is 
assumed to be the last year of the development period). 

2. Launch pad constraints, i. e. , maximum rate allowed. 

a. MISSION REQUIREMENTS AND COSTS . 

A. Mission Name. Priority. Launch Years, and the Number of Launches in 
Each Year 

B. Costs 

1. Development — including the number of years over which the develop- 
ment cost is distributed and the year in which development starts. 

2. Recurring and four year distribution. 

3. Sustaining and the number of years after the last launch date in which 
sustaining costs are incurred. 

4. Miscellaneous — such as run-out costs. Distribution is input. 

C. Requirements 

1. Velocity requirement. 

2. Launch site constraint, if any. 

3. Payload weight either per launch or total weight to be delivered iii less 
than NTRIP launches (input) i. e. modularization. 

4. Return payload weight. 

5. Stabilization requirement, if any. 

6. Manrating requirement, if any. 

7. Number of restarts required. 

8. Number of days a reusable upper stage is required to complete mission 
(if appropriate). 
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Table 4-1 (Cotit. ) 


m. BUDGET LEVEL DATA 

1. Desired funding level for each year. 

2. Calendar years for smoothing. 

3. Total fixed overhead costs for each year. 

4. Inflation rate. 

5. Program constraints. 

6. Smoothing constraint options. 
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non-computer oriented user. Not all data listed are necessary for a successful run. 

In fact the only required input is the following: 

« Initial launch plus reference year and mission model duration 

• At least one stage with associated recurring, development, and sustaining 
costs, and development year start and duration 

• At least one vehicle, whose component stages have all been input, and 
whose performance, coefficients are avilable (vehicle maybe composed 
of only one stage) 

• At least one mission with its launch rate and payload and velocity reqidre- 
• ments (no cost data need be input for a mission) 

• The desired funding level for each year and the calendar years over which 
smoothing is desired 

All input not on the above basic list is included in order to generate complete and 
realistic solutions. 

Recurring costs are input in terms of stage or integration costs. Any recurring costs 
associated with a specific launch pad are input under the associated booster recurring 
costs. Vehicle recurring costs are then computed by the program as the sum of all' 
component stage recurring costs plus any applicable integration recurring costs. 

Annual costs are quite complex in nature and hence require a detailed format for intro 
duction into the program. For instance, fixed launch pad costs are launch complex 
oriented while annual costs are stage oriented. Annual costs are further complicated 
by the fact that a second pad does not require the same number of people to maintain 
it as' the first pad. Discipline personnel are not fully utilized with only one pad and 
thus need not be duplicated for the second pad. However, other workers cannot main- 
tain two pads at once, so they must be duplicated for the second pad. 

Fixed costs may be entered for shared cost groups (families) or individual stages for 
hardware and/or launch pad expenditures. Launch facility expenditures due to integra 
tion between two stages are, in general, negligible, so this category was eliminated 
from consideration. 
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The above data is input to DATINS and CAPBLI, both subroutines of ASIGNS.' Essen- 
tially only the budget levels, smoothing intervals and program constraints are now 
input directly to the SMOTHS subroutine. An inflation factor may be applied to all 
costs to be spent in the future. The anticipated budget appropriation or desired level 
of spending is not adjusted by the inflation factor since inflation does not directly 
affect the budget level. When SMOTHS is used, Ihe budget level can be input year- 
by-year to reflect any growth or decline which might result from variation(s) in 
economic conditions. Thus, the impact of various economic effects can be defined 
and included in the model. 

4. 3 program options 

The options available to the analyst are of two types: (1) automatically determined 
by the program from the data input and (2) specified directly by the user. The 
deterministic option explained in Section 4. 2 is of type 1 since the program automatically 
bypasses all probabilistic calculations if all costs are deterministic. Rate effects on 
recurring costs are also ignored if no learning curve percentages are input. Some 
default options include the automatic distribution of launch vehicle recurring costs 
unless overridden by input to the variable ALPI, the automatic input of zero to most 
applicable budget items unless overridden by actual input, and the automatic use of 
the extension and acceleration options in the smoothing section unless FALSE is 
specified for the variables EXT or ACCL respectively. If NSOL (the number of solutions 
to be output in ascending order of total program cost), is input as zero^ one optimal 
solution will still be found. 

There are four major options which must be specified by the user — LP, MOS, NOPT, 
and NU. 

4.3.1 LP Option 

The first such option is the code for logic printout. In a test run code LP = 2 
should be used so that the internal logic may be checked for accuracy. Many lines 
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of output are required, however, so this option should not be used in general. If 
LP = 1, suboptimal solutions may be traced in the branch and bound logic and the 
optimal solution justified step by step. Thus, reasons for selection or non-selection 
of a program element in an assignment may be determined in detail if desired. LP = 0 
is the normal mode for production runs. Only final soluticms and characteristics of 
these solutions are output under this last option. 

4. 3, 2 MOS Option 

In order to accommodate some of the various uses which the analyst may have for the 
model, four options are made available to the user. On the first data card, the user 
specifies which option he desires by an appropriate value for MOS (method of solution). 

MOS = 0 Optimize launch vehicle assignment and smooth the resulting 
budget within constraints input to SMOTHS 
MOS = 1 Input specific launch vehicle assignment and smooth the 
resulting budget 

MOS = 2 Optimize launch vehicle assignment and output associated costs 
by year and program (do not smooth budget) 

MOS = 3 Input specific launch vehicle assignment and print out associated 
costs by year and program 

Thus the optimal assignment program without smoolhing is available using MOS = 2, 
the smoothing program alone using MOS = 1 and the integrated program using MOS = 0. 
MOS = 3 is useful in testing assignments derived from outside sources. Total cost 
distributions are then available for these assignments which may be compared to 
previously found optimal assignments. 

4. 3. 3 NOPT Option 

The mission/vehicle compatibility screen may be in one of three forms. The basic 
screen (NOPT = 1) consists of first looking to see if there is an a priori vehicle 
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assignment. K there is one', all other vehicles are excluded from consideration for 
that mission. . 

If there is no such pre-assignment, the payload capability of' the vehicle is compared 
to the payload desired for each mission at the required characteristic velocity. Mod- 
ularization is taken into consideration in determining whether the launch vehicle can 
or cannot accomplish the mission. The availability of each vehicle for a particular 
mission is determined later in subroutine AVAILI, where a final compatibility matrix 
is output. 

If NOPT = 2 is specified, the basic screen above is applied to any vehicle input 
directly and to all vehicles formed in the stage -matching screen performed in sub- 
routine MATEI. 

If NOPT = -3 is specified, the basic screen is augmented by tests on the stabilization, 
man-rating and other requirements input on the mission card. If NOPT is not specified 
as 2 or 3, then the basic screen is the default option. 

4. 3. 4 NU Option 

NU, the number of reusable units to be purchased, is zero if the stage is expendable. 
However, if the stage is reusable then either a positive number is input to NU and this 
number is used directly by the program throughout all iterations, or a negative number 
is input to NU and then the program uses this estimate for the first iteration but-calcu- 
lates its own estimate based on actual usage for succeeding estimates. The program 
estimate is based on turn-around-time, amortization lifetime, and mission use time, 
as appropriate. The estimate is calculated in subroutine REUSEI (the logic flow diagram 
is in Appendix C, Vol. 2). 

Other options such as using the Beta distribution or an alternative input distribution 
for any development' cost are explained in the comment section of Appendix A (Vol. 2) 

. Input Bequirements, 
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4. 4 SMOOTHING CONSTRAINTS 


Constraints, are input directly to SMOTHS for missions and for miscellaneous programs 
having no associated, launches. They are keyed according to the following table where; 


KOBE 

NPEOG 

KPROG 

CS 


the type of constraint by key number (see Table 4-2) 
N = the constrained program reference number 
K = the constraining program reference number 
associated real number constant 


Table 4-2 

KEY TO PROGRAM CONSTRAINTS^^^ ' 


KOBE 


1 

START^. > END„ + CS 
N K 

2 

END„ + CS < STARTs 
N K 

3 

START^ = CS 

4 

ENDj^ = CS 

5 

DEV. DURATlONjj = CS (FIXED DURATION) 

6 

LAUNCH DATE^^ + CS < LAUNCH DATE^ 
N K 

7 

LAUNCH DATE„ CS 
N 

8 

NO CHANGES ALLOWED 

9 

STARTj^ 2 CS 

10 

LAUNCH DATE^ > CS 

11 

ENDj^ + CS < LAUNCH DATEj^ 


(a) START and END refer to development. 


Input program data must satisfy the input constraints to ensure a correct output from 
SMOTHS. Any violations in input data are printed out before "smoothing” begins so 
that the user is aware of the condition. The program will continue even if violations 
occur since in many cases the violations are corrected by the "shifting" process. 


4-15 




Costs associated with launch vehicles in the optimal solution are automatically con- 
strained in MASTEB. KOBE 11 is used to ensure that all development programs 
selected by ASIGNS in the optimal solution end before the component being developed 
is to be launched. Thus, SMOTHS is automatically constrained so that the optimal 
vehicle assignment input to SMOTHS is still a feasible candidate assignment after 
SMOTHS is complete- Whether the assignment input to SMOTHS is still optimal 
depends on which variables have been "shifted'* by SMOTHS. If key variables have 
been changed, ASIGNS is called to again determine the optimal assignment. Depending 
on the, effect of the "shift" changes, this new optimal assignment can be the same as 
the previous assignment or it can be different. 

Even if the shift constraints Imposed on SMOTHS prevent the smoothing of an optimal 
assignment into one which is less than the budget level, the program will still output 
an optimum assignment; however, in this case the root-ihean-square cost difference 
between the actual spending level and the imposed budget level will be larger than if 
less constraints had been imposed. 

4.5 OUTPUT 

A sample output is presented in detail in Appendix B (Vol. 2) and some of the analytic 
relationships to be output were discussed in section 3.4. This present section will 
briefly cover the output from this program and section 4. 6 will discuss how this output 
may be utilized. 

First, all the input data are output for reference, including data computed by the pro- 
gram which will be input to the ASIGNS algorithm. Both input modal values and com- 
puted expected values are output whenever appropriate. Then the optimal assignment 
is output listing each mission and the assigned optimal vehicle, along with total mission 
model cost. If NSOL is greater than one, each assignment in ascending order of 
expected total cost is output until NSOL solutions have been found. For each assign- 
ment, the log-normal distribution describing the uncertainties associated with its total 
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cost is output along with its modal (most likely) value and 50% uncertainty interval. 

Each assignment is compared to each preceding one in order to determine the probability 
that it will cost more than the one preceding, given a definite correlation between 
assignments. 

Input to SMOTHS is output automatically as it appears on the data card. "Average" 
recurring cost data for each vehicle in the optimal assignment is computed in VEHRC. 
This cost is determined by totaling the actual recurring costs of all program elements 
associated with each vehicle over the entire mission duration and then dividing by the 
total number of vehicles used throughout the mission model. The constraints input to 
the program and those calculated in MASTER are output for reference. Any violations 
to these constraints in the input data are noted. Finally the cost data comprising the 
optimal assignment that is input to SMOTHS is output by program and type and also by 
year. A plot showing expected spending by year and desired spending level by year 
follows. The most likely (modal) spending level by year and the upper bound on a 50% 
confidence interval are also included on the plot. 

The program then smooths this input data and outputs the final result in the same form 
as it did the input data. Launch vehicle requirements by year are output using the 
smoothed data. At this point the program either terminates because an optimal smoothed 
assignment has been found or else it returns to ASIGNS and outputs the new data which 
will be used in the algorithm. The output cycle then continues as explained above until 
an optimal solution has been found. 

4. 6 PRODUCTION RUNS AND SENSITIVITY ANALYSES 

Extensive production runs have been made using the deterministic version of this model 
to checkout logic on a large-scale basis. As budget levels were allowed to vary 
parametrically, the mission models waried accordingly until a "smoothed" solution 
could be found for each case. The type of mission to be included in any future mission 
profile can be specified by the analyst; however, some mission types may be shown to 
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be too expensive for realistic budget levels -through use of this program. Since the 
least cost total space program which accomplishes the mission model is always selected 
by the program for budget analysis, the mission model is the only variable which needs 
to be modified as the budget level varies. 

Logic checkout runs for the probabilistic model were similar in nature to the production 
runs made earlier. Due to the fact that the probabilistic model optimizes on expected 
costs, while the deterministic model optimizes on most likely costs, the solutions 
from each model will not necessarily be the same. Therefore in analyzing a program 
it may be desirable to make runs on both the statistical and deterministic models in 
order to compare solutions and identify any dominant factors that are influencing the 
selection of the optimal assignment. 

As an example Fig. 4-5 demonstrates one sensitivity tradeoff which may be made using 
output from the model. One NASA mission profile was fixed and runs were made using 
various projected reusable vehicles and several expendable vehicles as candidates for 
assignment. One set of runs were made on the deterministic model, varying only the 
number of launches over a 10-year period. A similar set was then completed on the 
probabilisitc model. For the example input data used in this case, the least cost 
vehicle for this mission model consisted of an expendable lower and second stage with 
a reusable upper stage if the launch rate was less than 110 over 10 years for the 
deterministic model. For the probabilistic model the critical launch rate was found 
to be 190 over 10 years. 

That is, the transition to the optimal assignment of an (ultimately) less costly candidate 
system (i. e. , from expendable vehicle 1 + a reusable spacecraft to the partially reusable 
launch vehicle 2) occurred at line A (110 launches/lO yr) for the deterministic case, 
and at line B (190 launches/l6 yr) for the probabilistic case. This variation in transi- 
tion occurs because the uncertainties in the costs of the partially reusable system aie 
large, in comparison to the relative certainty of costs associated with an upgraded 
existing vehicle. 
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Fig. 4-5 Optimal Launch Vehicle Sensitivity Analysis 




Also as indicated by the two curves in this figure, for positively skewed distributions, 
determined to be typical of advanced technology space programs based on the analysis 
in Section 2, the expected (mean) value will be greater than the model value. This 
difference derives from the fact that in high technology programs, which typically have 
cost growth factors of two or more, there are a significant nunaber of programs with 
quite high cost growths. This situation produces a long tail on the distribution and an 
expected value greater than the mode. The expected value is a more realistic cost 
estimate than -the model value for these advanced programs since it corresponds to the 
center of gravity of the distribution. 

The capability, of the probabilistic model to output expected value outcomes and also 
outcomes that can be quantified to any degree of certainly is indicative of the effect- 
iveness of this new tool. The necessity for more effectively handling the cost un- 
certainties in advanced technology systems by using a probabilistic approach has 
received recent emphasis (Bef. 18). 

As indicated/ the analysis illustrated in Fig. ,4-5 was made with a fixed mission model. 

By varying this mission model, sensitivities to mission requirements may also be 
determined. These examples are some of the many sensitivity analyses and other 
tradeoffs which can be made using the probabilistic computer model. . 

The extensive flexibilities inherent in this model, its capability to quantitatively evaluate 
the uncertainties known to be present in advanced space program costs, and the ability 
to quantify outcomes to any degree of certainty provide the user with a unique evaluation 
tool. 
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Section 5 

CONCLUDING REMARKS 

5 . 1 PROBABILISTIC MODEL DEVELOPMENT 

Prior analyses of historical data and analyses performed during this study have clearly 
shown tiiat there are predictable undertainties in data applicable to advanced develop- 
ment space systems. This variability is particularly evident for costs in areas that 
have technological risk and under economic conditions which can change during a 
system life cycle. 

Under these conditions of variability, a probabilistic model is an essential tool for 
providii^ quantitative evaluations for space systems and over-all programs. The 
need for a probabilistic approach to problems of this type has recently been emphasized 
(Ref. 18). This present work provides NASA with a unique and advanced tool in this 
problem area. 

Building on a previously developed deterministic model, durii^ the present study 
effort the' space program optimal ass:ignment and budget smoothing model was con- 
verted to a probabilistic model. Check-out runs with this model have been performed; 

5.2 SUMMARY OF RESULTS 

The followii^ include key results of the study and outline significant capabilities of the 
probabilistic computer model. Details are provided in the body of this report. 

5.2.1 Primary Result 

The development of a comprehensive space program evaluation tool is a primary 
•result of this study. Using a significantly modified branch-and-bound technique for 
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accelerated search., the model quantitatively evaluates all combinations of launch 
vehicles and other interrelated space program elements to output a global optimum, 
least cost total space program over a one to 20 year period. The model retains the 
capability for deterministic solutions but adds the new, powerful dimension of prob- 
abilistic evaluations and sensitivity analyses, which quantify the cost uncertainties 
known to exist in high technology, advanced space programs. Based on the statistical 
input, the model can provide output data quantified to any degree of certainty. 

5.2.2 Data Analysis 

The analysis of historical data accomplished the following: 

• Provided a basis for analytic solution based on probability distributions 
rather than random number methods. This reduced solution complexity 
and retained short program run times. 

• Identified the log-normal distribution as the most appropriate type for 
advanced systems havii^ technology -risk. 

• Provided preliminary values for statistical input parameters. 

5.2.3 System Performance 

The model handles' launch vehicle and related element physical characteristics, 
functional performance, and time availability. 

5.2.4 Flexibility 

Multiple options for data input, internal analyses and output options provide flexible 
adaption of the model to the needs of the analyst. 

5. 2. 5 Range of Problems Solved 

The model can be applied to a broad range of space program evaluations. These 
extend from macro-problems that evaluate various options of total space programs, 
to intermediate problems which analyze selected categories of space programs (e. g. , 
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optimizing the scientific, exploratory, service satellite category within a total space 
program), and to micro-problems (e.g. , determining the cost optimal subsystem 
among several alternates for a given space vehicle). 

5.2.6 Production Use 

The model has retained storage minimizing features and short run times. These 
permit program use in a multi-user computer system. 

5; 2. 7 Optimal Assignment and Budget Smoothing 

The model retains the capability for optimal assignment of program elements. These 
assignments can be smoothed between parameterized, year-by-year bu^et constraints 
and support bases, and imder varying external economic conditions including growth 
and inflation. 

5. 2. 8 Growth Potential 

The model is structured with independence between subroutines whenever feasible. 

This independence permits the addition of subroutines as required to extend its capa- 
bility with minimum interaction. For example orbit-to -orbit maneuvering capability 
can be easily added to reflect characteristics of increased importance of this capa- 
bility in future systems. 

Historical data analysis has provided preliminary values for statistical parameters 
for initial model use. Further analysis is desirable to more firmly quantify statistical 
parameters including cost growth, particularly in the upper tail region, and correla- 
tion between variables. 


Multi-dimensional output plots which bound certainty regions will provide quicker 
reaction capability and more lucid output. The use of cathode-ray tube output display 
will provide increased capability in both of these areas. 



5. 2. 9 Versatility 

The model provides for analyses of expehdahles, partially reusables, and fully re- 
usables within a total program mix. Evaluations and sensitivity trade-offs include the 
effects of uncertainty in all such analyses. 

5.2.10 Applications 

Because of its versatility this unique tool has broad application to diverse space pro- 
gram evaluation and sensitivity analyses. Two model applications which supplement 
its primary use are suggested. 

• Since output can be bounded by certainty regions based on historical data, 
more realistic outcomes are available for program estimates, evaluations, 
and tradeoffs. One use can be the detection of "buy-in's" and a basis for 
better quantified program control in the development of advanced systems. 

• The identification of new space program directions that link.the space 
program to. substantial national requirements , :and that can develop sub- 
stantial growth and show a profit over the long term. 

In the present study the developed methodology and the optimal assignment model applies 
to space systems. However, the technique can be readily applied to diverse optimal 
combinatorial problems by particularizing the parameters to the new problem. 
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